{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ec4088c",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/integrations/vector_db/milvus.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb798603",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"UpTrain\">\n",
    "  </a>\n",
    "</h1>\n",
    "\n",
    "<div style=\"text-align: center;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38965fdc",
   "metadata": {},
   "source": [
    "## Building RAG Pipeline using Milvus and Evaluating Results using UpTrain\n",
    "\n",
    "### Overview\n",
    "\n",
    "RAG is a technique that is used to increase the accuracy of an LLM's response using information retrieved from an external source.\n",
    "\n",
    "In this tutorial, we will walk through the use of [Milvus](https://milvus.io/) (an Open-Source Vector Database) and [UpTrain](https://uptrain.ai/) (an Open-Source LLM Evaluation Tool) to walk through the key components of a RAG Pipeline.\n",
    "\n",
    "A Vector Database can help implement similarity search on your document (afterall your knowledge base could be pretty large, and you would just want to pass the relevant information to an LLM)\n",
    "\n",
    "There are multiple parts in a RAG pipeline and a failure can occur at any step. Tools like UpTrain can help you analyze LLM generated responses using over 20+ pre-built evaluations (covering use cases such as response quality, context awareness, code related evals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02148aff",
   "metadata": {},
   "source": [
    "### Install Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "90563474",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pymilvus in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (2.4.0)\n",
      "Requirement already satisfied: uptrain in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (0.6.10.post1)\n",
      "Requirement already satisfied: milvus-model in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (0.1.0)\n",
      "Requirement already satisfied: setuptools>=67 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (69.2.0)\n",
      "Requirement already satisfied: grpcio<=1.60.0,>=1.49.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (1.60.0)\n",
      "Requirement already satisfied: protobuf>=3.20.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (3.20.0)\n",
      "Requirement already satisfied: environs<=9.5.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (9.5.0)\n",
      "Requirement already satisfied: ujson>=2.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (5.4.0)\n",
      "Requirement already satisfied: pandas>=1.2.4 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (2.2.0)\n",
      "Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (2.31.0)\n",
      "Requirement already satisfied: minio>=7.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (7.2.5)\n",
      "Requirement already satisfied: pyarrow>=12.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (15.0.0)\n",
      "Requirement already satisfied: azure-storage-blob in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (12.19.1)\n",
      "Requirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pymilvus) (1.12.0)\n",
      "Requirement already satisfied: pydantic in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (2.6.4)\n",
      "Requirement already satisfied: pydantic-settings in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (2.2.1)\n",
      "Requirement already satisfied: loguru in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (0.7.2)\n",
      "Requirement already satisfied: lazy-loader in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (0.3)\n",
      "Requirement already satisfied: networkx in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (3.2.1)\n",
      "Requirement already satisfied: polars>=0.18 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (0.20.7)\n",
      "Requirement already satisfied: numpy>=1.23.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (1.26.4)\n",
      "Requirement already satisfied: httpx>=0.24.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (0.25.2)\n",
      "Requirement already satisfied: plotly>=5.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (5.19.0)\n",
      "Requirement already satisfied: aiolimiter>=1.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (1.1.0)\n",
      "Requirement already satisfied: openai>=1.6.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (1.14.1)\n",
      "Requirement already satisfied: fsspec in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (2023.10.0)\n",
      "Requirement already satisfied: litellm in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (1.26.3)\n",
      "Requirement already satisfied: pyyaml in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (6.0.1)\n",
      "Requirement already satisfied: json5 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from uptrain) (0.9.22)\n",
      "Requirement already satisfied: sentence-transformers in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (2.3.1)\n",
      "Requirement already satisfied: FlagEmbedding>=1.2.2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (1.2.8)\n",
      "Requirement already satisfied: nltk in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (3.8.1)\n",
      "Requirement already satisfied: transformers>=4.33.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (4.37.2)\n",
      "Requirement already satisfied: jieba in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (0.42.1)\n",
      "Requirement already satisfied: konlpy in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (0.6.0)\n",
      "Requirement already satisfied: mecab-python3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (1.0.8)\n",
      "Requirement already satisfied: unidic-lite in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from milvus-model) (1.0.8)\n",
      "Requirement already satisfied: marshmallow>=3.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from environs<=9.5.0->pymilvus) (3.20.2)\n",
      "Requirement already satisfied: python-dotenv in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from environs<=9.5.0->pymilvus) (1.0.1)\n",
      "Requirement already satisfied: torch>=1.6.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from FlagEmbedding>=1.2.2->milvus-model) (2.2.0)\n",
      "Requirement already satisfied: datasets in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from FlagEmbedding>=1.2.2->milvus-model) (2.18.0)\n",
      "Requirement already satisfied: accelerate>=0.20.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from FlagEmbedding>=1.2.2->milvus-model) (0.28.0)\n",
      "Requirement already satisfied: anyio in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (4.2.0)\n",
      "Requirement already satisfied: certifi in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.0.2)\n",
      "Requirement already satisfied: idna in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (3.6)\n",
      "Requirement already satisfied: sniffio in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpx>=0.24.1->uptrain) (1.3.0)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from httpcore==1.*->httpx>=0.24.1->uptrain) (0.14.0)\n",
      "Requirement already satisfied: urllib3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from minio>=7.0.0->pymilvus) (2.2.0)\n",
      "Requirement already satisfied: argon2-cffi in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from minio>=7.0.0->pymilvus) (23.1.0)\n",
      "Requirement already satisfied: pycryptodome in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from minio>=7.0.0->pymilvus) (3.20.0)\n",
      "Requirement already satisfied: typing-extensions in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from minio>=7.0.0->pymilvus) (4.9.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (1.9.0)\n",
      "Requirement already satisfied: tqdm>4 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from openai>=1.6.1->uptrain) (4.66.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/shreyanshjain/Library/Python/3.11/lib/python/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pandas>=1.2.4->pymilvus) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pandas>=1.2.4->pymilvus) (2023.4)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from plotly>=5.0.0->uptrain) (8.2.3)\n",
      "Requirement already satisfied: packaging in /Users/shreyanshjain/Library/Python/3.11/lib/python/site-packages (from plotly>=5.0.0->uptrain) (23.2)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pydantic->uptrain) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.16.3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pydantic->uptrain) (2.16.3)\n",
      "Requirement already satisfied: filelock in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from transformers>=4.33.0->milvus-model) (3.13.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from transformers>=4.33.0->milvus-model) (0.21.4)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from transformers>=4.33.0->milvus-model) (2023.12.25)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from transformers>=4.33.0->milvus-model) (0.15.2)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from transformers>=4.33.0->milvus-model) (0.4.2)\n",
      "Requirement already satisfied: azure-core<2.0.0,>=1.28.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from azure-storage-blob->pymilvus) (1.30.1)\n",
      "Requirement already satisfied: cryptography>=2.1.4 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from azure-storage-blob->pymilvus) (42.0.5)\n",
      "Requirement already satisfied: isodate>=0.6.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from azure-storage-blob->pymilvus) (0.6.1)\n",
      "Requirement already satisfied: JPype1>=0.7.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from konlpy->milvus-model) (1.5.0)\n",
      "Requirement already satisfied: lxml>=4.1.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from konlpy->milvus-model) (4.9.4)\n",
      "Requirement already satisfied: aiohttp in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from litellm->uptrain) (3.9.3)\n",
      "Requirement already satisfied: click in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from litellm->uptrain) (8.1.7)\n",
      "Requirement already satisfied: importlib-metadata>=6.8.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from litellm->uptrain) (6.11.0)\n",
      "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from litellm->uptrain) (3.1.3)\n",
      "Requirement already satisfied: tiktoken>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from litellm->uptrain) (0.5.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from requests->pymilvus) (3.3.2)\n",
      "Requirement already satisfied: joblib in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from nltk->milvus-model) (1.3.2)\n",
      "Requirement already satisfied: scikit-learn in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from sentence-transformers->milvus-model) (1.4.0)\n",
      "Requirement already satisfied: sentencepiece in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from sentence-transformers->milvus-model) (0.1.99)\n",
      "Requirement already satisfied: Pillow in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from sentence-transformers->milvus-model) (10.2.0)\n",
      "Requirement already satisfied: psutil in /Users/shreyanshjain/Library/Python/3.11/lib/python/site-packages (from accelerate>=0.20.1->FlagEmbedding>=1.2.2->milvus-model) (5.9.8)\n",
      "Requirement already satisfied: six>=1.11.0 in /Users/shreyanshjain/Library/Python/3.11/lib/python/site-packages (from azure-core<2.0.0,>=1.28.0->azure-storage-blob->pymilvus) (1.16.0)\n",
      "Requirement already satisfied: cffi>=1.12 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from cryptography>=2.1.4->azure-storage-blob->pymilvus) (1.16.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm->uptrain) (3.17.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm->uptrain) (2.1.5)\n",
      "Requirement already satisfied: sympy in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from torch>=1.6.0->FlagEmbedding>=1.2.2->milvus-model) (1.12)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from aiohttp->litellm->uptrain) (1.9.4)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from argon2-cffi->minio>=7.0.0->pymilvus) (21.2.0)\n",
      "Requirement already satisfied: pyarrow-hotfix in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from datasets->FlagEmbedding>=1.2.2->milvus-model) (0.6)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from datasets->FlagEmbedding>=1.2.2->milvus-model) (0.3.8)\n",
      "Requirement already satisfied: xxhash in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from datasets->FlagEmbedding>=1.2.2->milvus-model) (3.4.1)\n",
      "Requirement already satisfied: multiprocess in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from datasets->FlagEmbedding>=1.2.2->milvus-model) (0.70.16)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from scikit-learn->sentence-transformers->milvus-model) (3.2.0)\n",
      "Requirement already satisfied: pycparser in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=2.1.4->azure-storage-blob->pymilvus) (2.21)\n",
      "Requirement already satisfied: mpmath>=0.19 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from sympy->torch>=1.6.0->FlagEmbedding>=1.2.2->milvus-model) (1.3.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install pymilvus uptrain milvus-model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6e0f43f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from openai import OpenAI\n",
    "import wikipedia\n",
    "import os\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe8609d5-9f27-4202-b0be-36db34412998",
   "metadata": {},
   "source": [
    "### Defining the data \n",
    "\n",
    "In this example we will be considering a case where user's questions are related to the 2020 Summer Olympics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1d00341f",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = [\n",
    "    \"How many events were there in the 2020 Summer Olympics\",\n",
    "    \"Who performed at the opening ceremony of the 2020 Summer Olympics?\",\n",
    "    \"Where did the 2020 Summer Olympics take place?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d81789",
   "metadata": {},
   "source": [
    "**Let's first define a knowledge base**\n",
    "\n",
    "In this example, we will be using a [Wikipedia page](https://en.wikipedia.org/wiki/2020_Summer_Olympics) about the 2022 Summer Olympics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ffa49b0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_data = wikipedia.page(\"2020 Summer Olympics\").content "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "748d87d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_data[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aad1aa61-9e1c-46c8-ae5e-61855df440f9",
   "metadata": {},
   "source": [
    "**Split Document into Chunks**\n",
    "\n",
    "For ease of retrieving information, we will split the knowledge base into smaller chunks.\n",
    "\n",
    "Here, we have splitted the document into chunks of sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "209d1cfb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021',\n",
       " 'Tokyo was selected as the host city during the 125th IOC Session in Buenos Aires, Argentina on 7 September 2013.Originally scheduled to take place from 24 July to 9 August 2020, the event was postponed to 2021 on 24 March 2020 due to the global COVID-19 pandemic, the first such instance in the history of the Olympic Games (previous games had been cancelled but not rescheduled)',\n",
       " 'However, the event retained the Tokyo 2020 branding for marketing purposes',\n",
       " 'It was largely held behind closed doors with no public spectators permitted due to the declaration of a state of emergency in the Greater Tokyo Area in response to the pandemic, the first and only Olympic Games to be held without official spectators',\n",
       " 'The Games were the most expensive ever, with total spending of over $20 billion.The Games were the fourth Olympic Games to be held in Japan, following the 1964 Summer Olympics (Tokyo), 1972 Winter Olympics (Sapporo), and 1998 Winter Olympics (Nagano)']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks = base_data.split('. ')\n",
    "\n",
    "# Let's Look at the first 5 chunks\n",
    "chunks[:5] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f45db027",
   "metadata": {},
   "source": [
    "### Create embeddings \n",
    "\n",
    "for the information present in chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "46503830-6ad5-493e-a629-152721e2d88e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import model\n",
    "\n",
    "# initialize using 'text-embedding-3-large'\n",
    "openai_ef = model.dense.OpenAIEmbeddingFunction(\n",
    "    model_name=\"text-embedding-3-large\", \n",
    "    dimensions=512 \n",
    ")\n",
    "\n",
    "context_embeddings = openai_ef(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "18e48cd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_embeddings_formatted = []\n",
    "\n",
    "for index in range(len(context_embeddings)):\n",
    "    context_embeddings_formatted.append(\n",
    "        {\n",
    "            \"id\": index,\n",
    "            \"vector\": context_embeddings[index]\n",
    "        }\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75903ca7",
   "metadata": {},
   "source": [
    "### Store these embeddings in Milvus Vector DB\n",
    "\n",
    "In this example we have hosted Milvus locally using Docker.\n",
    "\n",
    "You can check out this [documentation](https://milvus.io/docs/install_standalone-docker.md) to learn how to install Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a4226fcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import MilvusClient\n",
    "\n",
    "CLUSTER_ENDPOINT = \"http://localhost:19530/\"\n",
    "\n",
    "milvus_client = MilvusClient(\n",
    "    uri=CLUSTER_ENDPOINT,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c39f995f",
   "metadata": {},
   "source": [
    "We will now create a collection in Milvus to store the embeddings that we created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2c03bcb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "MILVUS_COLLECTION_NAME = \"UpTrain_Milvus\"\n",
    "\n",
    "milvus_client.create_collection(\n",
    "    collection_name=MILVUS_COLLECTION_NAME,\n",
    "    dimension=512\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f512604",
   "metadata": {},
   "source": [
    "We will now store the embeddings in the collection that we created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c22e5c61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'insert_count': 272,\n",
       " 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "milvus_client.insert(\n",
    "    collection_name=MILVUS_COLLECTION_NAME,\n",
    "    data=context_embeddings_formatted\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645be4b2",
   "metadata": {},
   "source": [
    "### Retrieve relevant chunk using embeddings stored in Milvus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2677ea7a",
   "metadata": {},
   "source": [
    "Create embeddings for list of questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d0bd04e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "questions_embeddings = openai_ef(questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9af9288e",
   "metadata": {},
   "source": [
    "Extract the chunks closest to the questions using the embeddings stored in Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3301c449",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = milvus_client.search(\n",
    "    collection_name= MILVUS_COLLECTION_NAME,\n",
    "    data= questions_embeddings,\n",
    "    limit=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b905526e",
   "metadata": {},
   "source": [
    "> **Note**: You can set the number of results you want to fetch (by using the argument `limit`), in this example we are fetching the best 2 result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7a735c5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'id': 171, 'distance': 0.6818565726280212, 'entity': {}},\n",
       "  {'id': 0, 'distance': 0.6670520305633545, 'entity': {}}],\n",
       " [{'id': 155, 'distance': 0.6477672457695007, 'entity': {}},\n",
       "  {'id': 153, 'distance': 0.6460253000259399, 'entity': {}}],\n",
       " [{'id': 0, 'distance': 0.7630819082260132, 'entity': {}},\n",
       "  {'id': 92, 'distance': 0.7080820798873901, 'entity': {}}]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22306f8",
   "metadata": {},
   "source": [
    "Now, let's extract the id of these matching vectors (i.e. id of the most similar chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "278f6153",
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_id_list = [[item['id'] for item in sublist] for sublist in res]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e29802c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[171, 0], [155, 153], [0, 92]]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector_id_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3dc044d",
   "metadata": {},
   "source": [
    "Using these ids we will retrieve the chunks most relevant to the question asked "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "314ff5ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_chunk = []\n",
    "\n",
    "for i in range(len(vector_id_list)):\n",
    "    retrieved_chunk.append([chunks[i] for i in vector_id_list[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0355bf90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['These five new sports were approved on 3 August 2016 by the IOC during the 129th IOC Session in Rio de Janeiro, Brazil, and were included in the sports program for 2020 only, bringing the total number of sports at the 2020 Olympics to 33.\\n\\n\\n=== Test events ===\\nA total of 56 test events were scheduled to take place in the run-up to the 2020 Olympics and Paralympics',\n",
       "  'The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021'],\n",
       " ['Emperor Naruhito formally opened the Games, and at the end of the torch relay the Olympic cauldron was lit by Japanese tennis player Naomi Osaka.For the first time in the 2020 Olympic Games, it was decided that one male and one female in each country would take turns holding flags and serve as two of them',\n",
       "  'One of these events was a concert held on 18 July, which featured J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated \"creatures\" based on illustrations \"embodying the thoughts and emotions of people from across the world\".The original plans for Nippon Festival included events such as Kabuki x Opera (a concert that would have featured stage actor Ichikawa Ebizō XI, opera singers Anna Pirozzi and Erwin Schrott, and the Tokyo Philharmonic Orchestra), an arts and culture festival focusing on disabilities, and a special two-day exhibition sumo tournament at the Ryōgoku Kokugikan shortly after the Olympics—which would have differed significantly from the traditional bi-monthly Honbasho tournaments, and featured special commentary in English and Japanese to help explain to spectators the customs and traditions of professional sumo, which are deeply rooted in the Shinto religion.\\n\\n\\n== The Games ==\\n\\n\\n=== Opening ceremony ===\\n\\nThe opening ceremony was held on 23 July 2021 in the Olympic Stadium in Tokyo'],\n",
       " ['The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021',\n",
       "  'The plans were made official on 1 November 2019 after Tokyo Governor Yuriko Koike accepted the IOC\\'s decision, despite her belief that the events should have remained in Tokyo.In general, as urban studies scholar Faure notes, \"The Tokyo 2020–2021 Games had a relatively moderate impact on the city, compared to previous cases such as Rio and Sochi, or cases in which a major Olympic park was built in Barcelona in 1992, and in Beijing in 2008']]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieved_chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "075b5244",
   "metadata": {},
   "source": [
    "### Generating responses using the question and context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "da042a53-4564-4057-9a60-9b57dffff6a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "open_ai_client = OpenAI(api_key = os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "def get_response(retrieved_context, question):\n",
    "    prompt = f\"\"\"\n",
    "        Context information is below.\n",
    "        ---------------------\n",
    "        {retrieved_context}\n",
    "        ---------------------\n",
    "        Given the context information and not prior knowledge, answer the query and relevant citations.\n",
    "        Query: {question}\n",
    "        Answer Format:\n",
    "        A python dictionary with 'response' and 'cited_context'\n",
    "        \"\"\"\n",
    "    \n",
    "    response = open_ai_client.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\", messages=[{\"role\": \"system\", \"content\": prompt}]\n",
    "        ).choices[0].message.content\n",
    "    # return response\n",
    "    return {'question': question, 'context': retrieved_context, 'response': json.loads(response)['response'], 'cited_context': json.loads(response)['cited_context']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b95ce2",
   "metadata": {},
   "source": [
    "### Define your dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7d548b6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "\n",
    "for i in range(len(questions)):\n",
    "    retrieved_context = retrieved_chunk[i]\n",
    "    retrieved_context = ' '.join(retrieved_context)\n",
    "    question = questions[i]    \n",
    "    data.append(get_response(retrieved_context, question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "75dc1b3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'question': 'How many events were there in the 2020 Summer Olympics',\n",
       "  'context': 'These five new sports were approved on 3 August 2016 by the IOC during the 129th IOC Session in Rio de Janeiro, Brazil, and were included in the sports program for 2020 only, bringing the total number of sports at the 2020 Olympics to 33.\\n\\n\\n=== Test events ===\\nA total of 56 test events were scheduled to take place in the run-up to the 2020 Olympics and Paralympics The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021',\n",
       "  'response': 'A total of 339 events were held in the 2020 Summer Olympics.',\n",
       "  'cited_context': 'A total of 56 test events were scheduled to take place in the run-up to the 2020 Olympics and Paralympics. The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021.'},\n",
       " {'question': 'Who performed at the opening ceremony of the 2020 Summer Olympics?',\n",
       "  'context': 'Emperor Naruhito formally opened the Games, and at the end of the torch relay the Olympic cauldron was lit by Japanese tennis player Naomi Osaka.For the first time in the 2020 Olympic Games, it was decided that one male and one female in each country would take turns holding flags and serve as two of them One of these events was a concert held on 18 July, which featured J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated \"creatures\" based on illustrations \"embodying the thoughts and emotions of people from across the world\".The original plans for Nippon Festival included events such as Kabuki x Opera (a concert that would have featured stage actor Ichikawa Ebizō XI, opera singers Anna Pirozzi and Erwin Schrott, and the Tokyo Philharmonic Orchestra), an arts and culture festival focusing on disabilities, and a special two-day exhibition sumo tournament at the Ryōgoku Kokugikan shortly after the Olympics—which would have differed significantly from the traditional bi-monthly Honbasho tournaments, and featured special commentary in English and Japanese to help explain to spectators the customs and traditions of professional sumo, which are deeply rooted in the Shinto religion.\\n\\n\\n== The Games ==\\n\\n\\n=== Opening ceremony ===\\n\\nThe opening ceremony was held on 23 July 2021 in the Olympic Stadium in Tokyo',\n",
       "  'response': \"The opening ceremony of the 2020 Summer Olympics featured performances by J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated 'creatures' based on illustrations embodying the thoughts and emotions of people from across the world.\",\n",
       "  'cited_context': \"For the first time in the 2020 Olympic Games, it was decided that one male and one female in each country would take turns holding flags and serve as two of them. One of these events was a concert held on 18 July, which featured J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated 'creatures' based on illustrations 'embodying the thoughts and emotions of people from across the world'.\"},\n",
       " {'question': 'Where did the 2020 Summer Olympics take place?',\n",
       "  'context': 'The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021 The plans were made official on 1 November 2019 after Tokyo Governor Yuriko Koike accepted the IOC\\'s decision, despite her belief that the events should have remained in Tokyo.In general, as urban studies scholar Faure notes, \"The Tokyo 2020–2021 Games had a relatively moderate impact on the city, compared to previous cases such as Rio and Sochi, or cases in which a major Olympic park was built in Barcelona in 1992, and in Beijing in 2008',\n",
       "  'response': 'The 2020 Summer Olympics took place in Tokyo, Japan.',\n",
       "  'cited_context': 'The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021.'}]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff1d8af0",
   "metadata": {},
   "source": [
    "### Performing Evaluations on RAG pipeline using UpTrain's Open-Source Software (OSS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a70c8f",
   "metadata": {},
   "source": [
    "UpTrain uses these 4 parameters to perform RCA on your RAG pipeline:\n",
    "\n",
    "|Parameter|Explanation|\n",
    "|--|--|\n",
    "|question|This is the query asked by your user.|\n",
    "|context|This is the context that you pass to an LLM (retrieved-context)|\n",
    "|response|The response generated by the LLM|\n",
    "|cited_context|The relevant portion of the retrieved context that the LLM cites along with the response.|\n",
    "\n",
    "You can refer to this [tutorial](https://github.com/uptrain-ai/uptrain/blob/main/examples/root_cause_analysis/rag_with_citation.ipynb) or [documentation](https://docs.uptrain.ai/tutorials/analyzing-failure-cases) to understand more about using UpTrain to evaluate a RAG pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "858cc724",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lazy_loader/__init__.py:185: RuntimeWarning: subpackages can technically be lazily loaded, but it causes the package to be eagerly loaded even if it is already lazily loaded.So, you probably shouldn't use subpackages with this lazy feature.\n",
      "  warnings.warn(msg, RuntimeWarning)\n",
      "100%|██████████| 3/3 [00:01<00:00,  1.62it/s]\n",
      "100%|██████████| 3/3 [00:04<00:00,  1.44s/it]\n",
      "100%|██████████| 3/3 [00:08<00:00,  2.83s/it]\n",
      "100%|██████████| 3/3 [00:06<00:00,  2.33s/it]\n",
      "100%|██████████| 3/3 [00:04<00:00,  1.48s/it]\n",
      "100%|██████████| 3/3 [00:02<00:00,  1.36it/s]\n",
      "100%|██████████| 3/3 [00:05<00:00,  1.96s/it]\n"
     ]
    }
   ],
   "source": [
    "from uptrain import RcaTemplate, EvalLLM\n",
    "\n",
    "import nest_asyncio\n",
    "nest_asyncio.apply()\n",
    "from loguru import logger\n",
    "logger.remove()\n",
    "\n",
    "\n",
    "eval_llm = EvalLLM(openai_api_key = os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "results = eval_llm.perform_root_cause_analysis(\n",
    "    data = data,\n",
    "    rca_template = RcaTemplate.RAG_WITH_CITATION\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "385736b4",
   "metadata": {},
   "source": [
    "### Let's look at some of the an evaluation generated by UpTrain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0ba149b",
   "metadata": {},
   "source": [
    "In this example, we can see that the LLM has hallucinated, by generating information which could not be verified by the information present in the context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a3bf69a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'How many events were there in the 2020 Summer Olympics',\n",
       " 'context': 'These five new sports were approved on 3 August 2016 by the IOC during the 129th IOC Session in Rio de Janeiro, Brazil, and were included in the sports program for 2020 only, bringing the total number of sports at the 2020 Olympics to 33.\\n\\n\\n=== Test events ===\\nA total of 56 test events were scheduled to take place in the run-up to the 2020 Olympics and Paralympics The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021',\n",
       " 'response': 'A total of 339 events were held in the 2020 Summer Olympics.',\n",
       " 'cited_context': 'A total of 56 test events were scheduled to take place in the run-up to the 2020 Olympics and Paralympics. The 2020 Summer Olympics, officially the Games of the XXXII Olympiad and officially branded as Tokyo 2020, were an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July 2021.',\n",
       " 'error_mode': 'Hallucinations',\n",
       " 'error_resolution_suggestion': 'Add instructions to your LLM to adher to the context provide - Try tipping',\n",
       " 'score_question_completeness': 1,\n",
       " 'score_valid_response': 1.0,\n",
       " 'explanation_valid_response': '{\\n    \"Reasoning\": \"The response \\'A total of 339 events were held in the 2020 Summer Olympics\\' provides the specific number of events held in the 2020 Summer Olympics. Therefore, the response does contain information relevant to the question.\",\\n    \"Choice\": \"A\"\\n}',\n",
       " 'score_context_relevance': 1.0,\n",
       " 'explanation_context_relevance': '{\\n    \"Reasoning\": \"The given context can answer the given question completely because it provides the specific information that there were 33 sports at the 2020 Olympics. This directly answers the query about the number of events in the 2020 Summer Olympics.\",\\n    \"Choice\": \"A\"\\n}',\n",
       " 'score_factual_accuracy': 0.0,\n",
       " 'explanation_factual_accuracy': '{\\n    \"Result\": [\\n        {\\n            \"Fact\": \"1. The 2020 Summer Olympics had a total of 339 events.\",\\n            \"Reasoning\": \"The context explicitly states that the total number of sports at the 2020 Olympics was 33, not 339 events. Hence, the fact cannot be verified by the context.\",\\n            \"Judgement\": \"no\"\\n        }\\n    ]\\n}',\n",
       " 'score_cited_context_relevance': 0.0,\n",
       " 'explanation_cited_context_relevance': '{\\n    \"Reasoning\": \"The given context does not contain specific information about the number of events in the 2020 Summer Olympics. It only mentions the scheduled test events and the dates of the Olympics. Therefore, the context does not provide enough information to answer the given query.\",\\n    \"Choice\": \"C\"\\n}',\n",
       " 'score_factual_accuracy_wrt_cited': 1.0,\n",
       " 'explanation_factual_accuracy_wrt_cited': '{\\n    \"Result\": [\\n        {\\n            \"Fact\": \"1. The 2020 Summer Olympics had a total of 339 events.\",\\n            \"Reasoning\": \"The context mentions that the 2020 Summer Olympics had a total of 339 events. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        }\\n    ]\\n}'}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec89551",
   "metadata": {},
   "source": [
    "In this example, we can see that the context doesn't have relevant information on where did the first Summer Olympics took place. Hence, the retrieval pipeline needs improvement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "53c28cce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'Who performed at the opening ceremony of the 2020 Summer Olympics?',\n",
       " 'context': 'Emperor Naruhito formally opened the Games, and at the end of the torch relay the Olympic cauldron was lit by Japanese tennis player Naomi Osaka.For the first time in the 2020 Olympic Games, it was decided that one male and one female in each country would take turns holding flags and serve as two of them One of these events was a concert held on 18 July, which featured J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated \"creatures\" based on illustrations \"embodying the thoughts and emotions of people from across the world\".The original plans for Nippon Festival included events such as Kabuki x Opera (a concert that would have featured stage actor Ichikawa Ebizō XI, opera singers Anna Pirozzi and Erwin Schrott, and the Tokyo Philharmonic Orchestra), an arts and culture festival focusing on disabilities, and a special two-day exhibition sumo tournament at the Ryōgoku Kokugikan shortly after the Olympics—which would have differed significantly from the traditional bi-monthly Honbasho tournaments, and featured special commentary in English and Japanese to help explain to spectators the customs and traditions of professional sumo, which are deeply rooted in the Shinto religion.\\n\\n\\n== The Games ==\\n\\n\\n=== Opening ceremony ===\\n\\nThe opening ceremony was held on 23 July 2021 in the Olympic Stadium in Tokyo',\n",
       " 'response': \"The opening ceremony of the 2020 Summer Olympics featured performances by J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated 'creatures' based on illustrations embodying the thoughts and emotions of people from across the world.\",\n",
       " 'cited_context': \"For the first time in the 2020 Olympic Games, it was decided that one male and one female in each country would take turns holding flags and serve as two of them. One of these events was a concert held on 18 July, which featured J-rock band Wanima, choreography by dancers Aio Yamada and Tuki Takamura, and the presentation of animated 'creatures' based on illustrations 'embodying the thoughts and emotions of people from across the world'.\",\n",
       " 'error_mode': 'Poor Retrieval',\n",
       " 'error_resolution_suggestion': 'Context Retrieval Pipeline needs improvement',\n",
       " 'score_question_completeness': 1,\n",
       " 'score_valid_response': 1.0,\n",
       " 'explanation_valid_response': '{\\n    \"Reasoning\": \"The response provides information about the performers at the opening ceremony of the 2020 Summer Olympics, including the J-rock band Wanima, choreographers Aio Yamada and Tuki Takamura, and the presentation of animated \\'creatures\\'. Therefore, the response does contain information relevant to the question.\",\\n    \"Choice\": \"A\"\\n}',\n",
       " 'score_context_relevance': 0.0,\n",
       " 'explanation_context_relevance': '{\\n    \"Reasoning\": \"The given context does not contain any information about the performer at the opening ceremony of the 2020 Summer Olympics. The context only mentions the opening ceremony date and some events related to the Olympics, but it does not provide any information about the performer. Hence, selected choice is C. The extracted context doesn\\'t contain any information to answer the given query.\",\\n    \"Choice\": \"C\"\\n}',\n",
       " 'score_factual_accuracy': 1.0,\n",
       " 'explanation_factual_accuracy': '{\\n    \"Result\": [\\n        {\\n            \"Fact\": \"1. The opening ceremony of the 2020 Summer Olympics featured performances by J-rock band Wanima.\",\\n            \"Reasoning\": \"The context explicitly mentions that the concert held on 18 July featured J-rock band Wanima. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        },\\n        {\\n            \"Fact\": \"2. The opening ceremony of the 2020 Summer Olympics featured choreography by dancers Aio Yamada and Tuki Takamura.\",\\n            \"Reasoning\": \"The context mentions that the concert held on 18 July featured choreography by dancers Aio Yamada and Tuki Takamura. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        },\\n        {\\n            \"Fact\": \"3. The opening ceremony of the 2020 Summer Olympics featured the presentation of animated \\'creatures\\' based on illustrations embodying the thoughts and emotions of people from across the world.\",\\n            \"Reasoning\": \"The context mentions that the concert held on 18 July featured the presentation of animated \\'creatures\\' based on illustrations embodying the thoughts and emotions of people from across the world. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        }\\n    ]\\n}',\n",
       " 'score_cited_context_relevance': 0.0,\n",
       " 'explanation_cited_context_relevance': '{\\n    \"Reasoning\": \"The given context does not contain any information about the performer at the opening ceremony of the 2020 Summer Olympics. It only mentions a concert with a J-rock band Wanima and other performers, but it does not specify that it was the opening ceremony. Therefore, the context does not provide relevant information to answer the query.\",\\n    \"Choice\": \"C\"\\n}',\n",
       " 'score_factual_accuracy_wrt_cited': 1.0,\n",
       " 'explanation_factual_accuracy_wrt_cited': '{\\n    \"Result\": [\\n        {\\n            \"Fact\": \"1. The opening ceremony of the 2020 Summer Olympics featured performances by J-rock band Wanima.\",\\n            \"Reasoning\": \"The context explicitly mentions that a concert held on 18 July featured J-rock band Wanima. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        },\\n        {\\n            \"Fact\": \"2. The opening ceremony of the 2020 Summer Olympics featured choreography by dancers Aio Yamada and Tuki Takamura.\",\\n            \"Reasoning\": \"The context explicitly mentions that a concert held on 18 July featured choreography by dancers Aio Yamada and Tuki Takamura. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        },\\n        {\\n            \"Fact\": \"3. The opening ceremony of the 2020 Summer Olympics featured the presentation of animated \\'creatures\\' based on illustrations embodying the thoughts and emotions of people from across the world.\",\\n            \"Reasoning\": \"The context explicitly mentions that a concert held on 18 July featured the presentation of animated \\'creatures\\' based on illustrations embodying the thoughts and emotions of people from across the world. Hence, the fact can be verified by the context.\",\\n            \"Judgement\": \"yes\"\\n        }\\n    ]\\n}'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[1]"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:light"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
